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V  ABSTRACT 

he  aim  of  this  performance  evaluation  is  twofold:  (I)  to  devise  benchmarking  strategies  for  and 
apply  benchmarking  methodologies  to  the  measurement  of  a  prototyped  database  system  in  multiple  back¬ 
end  configurations,  and  (2)  to  verify  the  performance  claims  as  projected  or  predicted  by  the  designer  and 
implementor  of  the  multi-backend  database  system  known  as  MBDS. 

Despite  the  limitation  of  the  backend  hardware,  the  benchmarking  experiments' have  proceeded  well, 
producing  startling  results  and  good  insights.  By  collecting  macroscopic  data  such  as  the  response  time  of 
the  request,  the  external  performance  measurements  of  MBDS  have  been  conducted.  The  performance 
evaluation  studies  verify  that  (a)  when  the  database  remains  the  same  the  response  time  of  a  request  can 
be  reduced  to  nearly  half,  if  the  number  of  backends  and  their  disks  is  doubled;  (b)  when  the  response  set 
of  a  request  doubles,  the  response  lime  of  the  query  remains  nearly  constant,  if  the  number  of  backends 
and  their  disks  is  doubled.  These  were  the  performance  claims  of  MBDS  as  predicted  by  its  designer  and 
implementor. 


*  The  work  reported  herein  is  supported  by  Contract  N000I4-84-WR-24058  from  the  OfTice  of  Naval 
Research  and  conducted  at  the  Laboratory  for  Database  Systems  Research,  Naval  Postgraduate 
School,  Monterey,  CA  93943. 
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1.  INTRODUCTION 

The  multi-backend  database  system  (MRDS)  is  a  database  system  designed  specifically  for  capacity 
growth  and  performance  enhancement.  MBDS  consists  of  two  or  more  minicomputers  and  their  dedicated 
disk  systems.  One  of  the  minicomputers  serves  as  a  controller  to  broadcast  the  requests  to  and  receive  the 
results  from  the  other  minicomputers,  which  are  configured  in  a  parallel  manner  and  are  termed  as  back¬ 
ends.  All  the  backend  minicomputers  are  identical,  and  run  identical  software.  The  database  is  evenly 
distributed  across  the  disk  drives  of  each  backend  by  way  of  a  duster-based  data  placement  algorithm 
unknown  to  the  user.  User  access  to  the  MBDS  is  accomplished  either  via  a  host  computer,  which  in  turn 
communicates  with  the  MBDS  controller,  or  with  the  MBDS  controller  directly.  Communication  between 
the  controller  and  backends  is  accomplished  using  a  broadcast  bus.  An  overview  of  the  system  architec¬ 
ture  is  given  in  Figure  1. 

There  are  two  basic  performance  claims  of  the  multi-backend  database  system,  which  have  been  pro¬ 
jected  in  the  original  design  goals  |lfsia81a,  HsiaSlb).  The  first  claim  states  that  if  the  database  sise 
remains  constant,  then  the  response  time  of  requests  processed  by  the  system  is  inversely  proportional  to 
the  multiplicity  of  backends.  This  claim  implies  that  by  increasing  the  number  of  backends  in  the  system 
and  by  replicating  the  system  software  on  the  new  backends,  MBDS  can  achieve  a  reciprocal  decrease  in 
the  response  time  for  the  same  requests.  The  second  claim  states  that  the  response  time  of  requests  is 
invariant  when  the  response  set  and  the  multiplicity  of  backends  increase  in  the  same  proportion.  This 
claim  implies  that  when  the  database  site  grows,  the  response  set  for  the  same  requests  will  grow.  By 
increasing  the  number  of  backends  accordingly,  MBDS  can  maintain  a  constant  response  time. 

In  this  paper  we  provide  a  preliminary  evaluation  of  the  validity  of  the  MBDS  performance  claims. 
The  main  focus  of  this  paper  is  on  the  external  performance  measurement  of  MBDS.  The  external  per¬ 
formance  measurement  evaluates  a  system  by  collecting  the  response  times  of  requests.  External  perfor¬ 
mance  measurement  is  a  macroscopic  evaluation  of  the  system.  Ingres,  Oracle,  and  the  Britton-Lee 
1DM/S00,  have  all  been  evaluated  using  external  performance  measurement  techniques  |Stra84,  Schi84). 

The  remainder  of  this  paper  is  organised  as  follows.  In  Section  2  we  provide  a  brief  overview  of  the 
multi-backend  database  system.  In  Section  S  we  discuss  the  general  testing  strategy  that  was  used  to 
evaluate  the  system.  In  Section  4  we  examine  the  evaluation  results.  Finally,  in  Section  8  we  conclude 
this  paper  and  summarise  the  results. 

3.  THE  MULTI-BACKEND  DATABASE  SYSTEM  (MBDS) 

The  current  hardware  configuration  of  MBDS  consists  of  a  VAX-1 1/780  (VMS  OS)  running  as  the 
controller  and  two  PDP-U/44s  (RSX-I1M  OS)  running  ns  backends.  Intercomputer  communication  is 
supported  by  three  parallel  communication  links  (PCL-llBs),  which  is  a  time-divisioned-multiplexed  bus. 
An  overview  of  MBDS  can  be  found  in  |Kerr82|.  The  implementation  efforts  are  documented  in  (He82, 
Boyn83b,  Demu84|.  MBDS  is  a  message-oriented  system  (see  [BoynSSa]).  In  a  message-oriented  system, 
each  process  corresponds  to  one  system  function.  These  processes,  then,  communicate  among  themselves 
by  passing  messages.  User  requests  are  passed  between  processes  as  messages.  The  message  paths  between 
processes  are  fixed  for  the  system.  The  MRDS  processes  are  created  at  the  start-up  time  and  exist 
throughout  the  entire  running  time  of  the  system. 

MBDS  provides  a  centralised  database  system  where  the  database  itself  is  evenly  distributed  across 
the  backend  processors.  Only  a  single  copy  of  the  database  is  stored.  The  underlying  data  model  for 
MBDS  is  the  attribute-based  data  model  |Ilsia70|.  The  attribute-based  data  model  stores  data  in  files  of 
records.  MBDS  stores  records  of  a  file  in  clusters.  A  c luster  is  a  group  of  records  such  that  every  record 
in  the  cluster  satisfies  the  same  set  of  attribute-value  pairs  or  ranges.  Thus,  a  file  is  divided  into  one  or 
more  clusters.  The  distribution  of  the  database  is  accomplished  using  a  cluster-based  data  placement 
algorithm. 

The  duster-based  data  placement  algorithm  is  arbitrated  and  managed  by  the  controller.  When  a 
new  cluster  is  defined,  the  backend  processor  notifies  the  controller.  The  controller  then  decides  which 
backend  will  insert  the  new  record.  Under  the  direction  of  the  controller,  the  chosen  baekend  will  con¬ 
tinue  to  insert  records  of  the  new  cluster,  until  the  baekend  processor  fills  a  block  of  secondary  memory 
storage.  When  this  occurs,  the  backend  processor  notifies  the  controller  that  the  block  is  full.  The  con¬ 
troller  then  directs  another  backend  for  the  insertion  of  new  records  of  the  cluster.  In  a  multiple-backend 
configuration,  the  controller  attempts  to  achieve  a  block-parallel-and-record-serial  operation  for  any  subse¬ 
quent  access  to  the  records  of  the  cluster. 
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Figure  1.  The  MDBS  Hardware  Organisation 
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Let's  trace  through  an  example.  Suppose  that  our  system  has  four  backend  processors,  the  average 
site  of  a  record  is  200  bytes,  and  the  sise  of  a  block  of  secondary  storage  memory  is  4K  (so,  each  block 
contains  approximately  20  records).  A  new  cluster  of  100  records,  say  C,  is  defined.  The  controller  picks 
say,  Backend  2,  for  inserting;  records  of  cluster  C.  Backend  S  will  insert  20  records  into  a  block  for  the 
cluster  C  under  the  direction  of  the  controller.  Then  the  controller  will  have  Backend  4  insert  records  of 
cluster  C.  After  Backend  4  has  inserted  20  records,  the  controller  will  cycle  to  Backend  1,  and  continue 
the  round-robin  process  until  all  100  records  are  placed  on  the  secondary  storage  blocks.  For  the  next  new 
cluster,  say,  C',  the  controller  will  then  pick  Backend  4,  since  Backend  S  is  the  last  backend  used  by  the 
previous  cluster  in  the  algorithm. 

3.  THE  BENCHMARK  STRATEGY 

In  this  section  we  analyse  the  basic  benchmark  strategy  for  the  preliminary  performance  evaluation 
of  the  multi-backend  database  system.  The  benchmark  strategy  focuses  on  collecting  macroscopic  meas¬ 
urements  on  the  systems  performance.  Macroscopic  measurements  correspond  to  the  external  performance 
measurement  of  the  system,  which  collects  the  response  time  of  requests  that  are  processed  by  the  system. 
To  adequately  conduct  the  external  performance  measurement  of  the  system,  software  was  developed  to 
collect  liming  information  and  data.  The  performance  software  was  bracketed  in  conditional  compilation 
statements  to  facilitate  an  easy  transition  between  a  testing  system  and  a  running  system. 

The  rest  of  this  section  is  organised  as  follows.  First,  we  give  a  high-level  description  of  the  test 
database  organisation  and  system  configurations  used  in  the  performance  evaluation.  Next,  we  examine 
the  request  set  used  to  collect  the  timings.  Finally,  we  review  the  relevant  tests  that  are  to  be  conducted, 
and  the  measurement  statistics  that  are  collected  and  calculated. 

8.1.  The  Test  Database  Organisation  and  Testing  Configurations 

The  test  database  was  constructed  using  a  record  site  of  200  bytes.  A  total  of  24  clusters  are 
defined  for  the  test  database.  The  virtual  and  physical  memory  limitations  of  each  backend  restricted  the 
database  size  to  a  maximum  of  1000  records  per  backend.  This  limitation,  coupled  with  the  need  to  verify 
the  two  performance  claims,  led  us  to  the  specification  of  three  different  system  configurations  for  the 
MBDS  performance  measurements.  Table  I  displays  the  configurations. 

Test  A  configures  MBDS  with  one  backend  and  one  thousand  records  in  the  test  database.  Test  B 
configures  MBDS  with  two  backends  and  one  thousand  records  split  evenly  between  the  backends.  The 
transition  from  Test  A  to  Test  B  is  used  to  verify  the  first  performance  claim  (see  Section  1).  Tests  A 
and  B  have  23  clusters  that  contain  40  records  and  one  cluster  that  contains  80  records.  In  Test  A,  all  of 
the  records  are  stored  on  the  single  backend.  In  Test  B,  each  backend  stores  20  records  for  the  first  23 
clusters  and  40  records  for  the  last  cluster. 

Test  C  also  configures  MBDS  with  two  backends,  but,  the  sise  of  the  database  is  doubled  to  two 
thousand  records.  The  transition  from  Test  A  to  Test  C  is  used  to  verify  the  second  performance  claim 
(see  Section  1).  Test  C  has  23  clusters  that  contain  80  records  each  and  one  cluster  that  contains  180 
records.  In  Test  C,  each  backend  stores  40  records  for  each  of  the  first  23  clusters  and  80  records  for  the 
last  cluster.  Notice  that  the  record  totals  per  cluster  per  backend  are  the  same  for  Test  A  and  Test  C. 

3.2.  The  Request  Set 

In  this  section  we  review  the  retrieve  requests  that  are  used  to  benchmark  MBDS.  The  retrievals, 
shown  in  Table  2,  are  a  mix  of  single  and  double  predicates.  There  are  two  directory  attributes  and 
thirty-one  non-directory  attributes  in  each  record.  The  directory  attributes,  INTEl  and  INTE2,  are 
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200K  bytes 
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1000 

400K  bytes 

Table  1.  The  Measurement  Configurations 
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integer-valued,  ud  in  wd  for  the  clutter  Miiitiot  tad  formation.  INTEl  it  defined  «iit|  S  attribute- 
value  raagea,  while  1NTE2  it  defined  uaiag  14  attribute- value  ranges.  The  uoa-directory  attributes  are 
uted  at  fillera  for  the  200-byte  record.  The  retrieve  request*  givea  ia  Table  2  are  specified  utiag  equality 
and  inequality  predicatee,  to  control  the  search  tpacc  when  accetting  the  database  records. 

In  Table  9  we  present  a  high-level  analysis  of  the  request  set  given  in  Table  2.  We  focus  on  specify¬ 
ing  two  characteristics  for  each  retrieve  request  in  the  request  set;  the  number  of  clusters  examined  by  the 
particular  retrieve  request  and  the  volume  of  the  database  information  that  is  retrieved.  The  values  in 
Table  9  apply  to  the  three  testing  configurations,  A,  R,  and  C,  with  one  exception.  The  numbers  in 
parenthesis  in  the  third  column  represent  the  number  of  records  retrieved  for  Test  C. 


S.S.  The  Measurement  Strategy,  Statistics  and  Limitations 

The  basic  measurement  statistics  used  in  the  performance  evaluation  of  MBDS  is  the  response  time 
of  request(s)  that  are  processed  by  the  database  system.  The  response  time  of  a  request  is  the  time 
between  the  initial  issuance  of  the  request  by  the  user  and  the  final  receipt  of  the  entire  request  set  for  the 
request.  The  response  times  are  collected  for  the  request  set  (see  Table  2)  for  each  of  the  three  configura¬ 
tions  (sec  Table  1).  Each  request  is  sent  a  total  of  ten  times  per  database  configuration.  The  response 
time  of  each  request  is  recorded.  We  determine  that  ten  repetitions  of  each  request  produce  an  acceptable 
standard  deviation.  Upon  completion  of  the  ten  re|»etitions  for  a  request,  we  calculate  the  mean  and  the 
standard  deviation  of  the  ten  response  times.  There  are  two  main  statistics  that  we  calculate  to  evaluate 
the  MBDS  performance  claims,  the  response-time  improvement  and  the  response-time  reduction. 

The  response- time  improvement  is  defined  to  be  the  percentage  improvement  in  the  response  time  of 
a  request,  when  the  request  is  executed  in  n  backends  as  opposed  to  one  backend  and  the  number  of 
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Retrieval  Request 


(INTEl  =  10)  or  (INTEl  =  290) 
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(iNT E2  =  <  500)  _ _ 

__I  (INTEl  =_<  1000) _ _ 

(INTEl  < 200)  or  (iNTEl  >=  801 ) 
(INTEl  =<  400)  or  (INTEl  >=  «6l) 
"  (INTEl  <=  201) 


(INTEl  <=  401) 
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Table  2.  The  Retrieval  Requests 


Request 

Number 

Number  of 
Clusters 
Examined 

Volume  of 

Database 

Retrieved 

1 

2 

2(4)  records 

2 

7 

25% 

9 

IS 

o 

4 

24 

100% 

5 

9 

40% 

6 

19 

80% 

7 

10 

20%  +  1(2)  record 

8 

15 

40%  •+  1(2)  record 

9 

19 

40%  -4  2(4)  records 

Table  9.  The  Number  of  Clusters  Examined  and  the 
Percent  of  the  Database  Retrieved 
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records  in  the  database  remain*  the  tame.  Equation  1  provide*  the  formula  used  to  calculate  the 
response-time  improvement  for  a  particular  request,  where  Configuration  Y  represents  n  backends  and 
Configuration  X  represents  one  backend.  The  response-time  improvement  is  calculated  for  the  configure-' 
tion  pair  (A,  B).  The  configuration  pair  (A,  B)  is  evaluated  for  the  retrieve  requests  (1)  through  (9)  (see 
Table  2). 

The  Response 
Time  of 
Con  f  tgurotiow  X 
The  Response 
Time  o/ 

Con/  iguration  Y 


Equation  1.  The  Response-Time-Improvement  Calculation 

The  reiponee-time  reduction  is  defined  to  be  the  reduction  in  response  time  of  a  request,  when  the 
request  is  executed  in  n  backends  containing  nx  number  of  records  as  opposed  to  one  backend  with  x 
number  of  records.  Equation  2  provides  the  formula  used  to  calculate  the  the  response-time  reduction  for 
a  particuic  retrieval  request,  where  configuration  X  represents  one  backend  with  x  records  and  configura-' 
tion  Z  represents  n  backends,  each  with  x  records.  The  response-time  reduction  is  calculated  for  the  confi¬ 
guration  pair  (A,  C),  for  the  retrieve  requests  (1)  through  (9). 

The  Response 
Time  of 
Configuration  Z 

The  Response 
Time  of 
Con  J  iguration  X 
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Response  Time  -  100%  * 
Reduction 


1  - 


The 

Response  Time  =  100%  * 
Improvement 


Equation  2.  The  Response-Time-Reduction  Calculation 

Finally,  we  examine  the  limitations  of  the  testing  strategy.  The  last  two  versions  of  MBDS  differ  in 
the  implementation  of  the  directory  tables.  The  newest  version  of  the  system,  called  Version  F,  imple¬ 
ments  the  directory  tables  on  the  secondary  storage.  The  previous  version,  called  Version  E,  stored  the 
directory  tables  in  the  primary  memory.  The  major  roadblock  that  we  have  encountered  in  the  perfor¬ 
mance  measurement  of  MBDS  has  been  the  hardware  limitations  of  the  backend  processors  (PDP-11/44). 
With  only  64K  of  virtual  memory  per  process  and  a  total  of  256K  physical  memory,  we  found  that  we 
could  not  increase  the  MBDS  system  parameters  to  permit  an  extensive  test  of  the  system  on  a  large 
database.  These  restrictions  have  forced  us  to  benchmark  the  primary-memory-based  directory  manage¬ 
ment  version  of  the  system  which,  excluding  the  directory  table  management  routines,  is  nevertheless 
equivalent  in  functionality  to  Version  F. 

4.  THE  BENCHMARKING  RESULTS 

In  this  section,  we  present  the  results  obtained  from  the  performance  measurement  of  MBDS.  In 
particular,  we  review  the  results  of  external  performance  measurement,  in  the  hope  of  verifying  the  MBDS 
performance  and  capacity  claims.  One  final  note,  the  units  of  measurement  presented  in  the  tables  of  this 
section  are  expressed  in  seconds. 

Table  4  provides  the  results  of  the  external  performance  measurement  of  MBDS.  There  are  three 
parts  to  Table  4.  Each  part  contains  the  mean  and  the  standard  deviation  of  the  response  times  for 
requests  (1)  through  (9),  which  are  outlined  in  Section  S.2.  The  three  parts  of  Table  4  represent  three  dif¬ 
ferent  configurations  of  the  MBDS  hardware  and  the  database  capacity.  The  first  part  has  configured 
MBDS  with  one  backend  and  the  database  with  1000  records  on  its  disk.  The  second  part  has  configured 
MBDS  with  two  backends,  with  the  database  of  1000  records,  split  evenly  between  the  disks  of  the  back¬ 
ends.  The  third  part  has  configured  MBDS  with  two  backends  and  with  a  database  doubled  of  2000 
records,  where  the  disk  of  each  backend  has  100  records. 
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Table  4.  The  Response  Time  Results 

Given  the  data  presented  in  Table  4,  we  can  now  attempt  to  verify  or  disprove  the  two  MBDS  per¬ 
formance  claims.  We  begin  by  calculating  the  response-time  improvement  for  the  nine  requests.  In  Table 
5  we  present  the  response-time  improvement  for  the  data  given  in  Table  4.  Notice  that  the  response-time 
improvement  is  lowest  for  request  (1),  which  represents  a  retrieval  of  two  records  of  the  database.  On  the 
other  hand,  the  response-lime  improvement  of  request  (4),  which  retrieves  all  of  the  database  information 
is  highest,  approaching  the  upper  bound  of  fifty  percent.  In  general,  we  find  that  the  response-time 
improvement  increases  as  the  number  of  records  retrieved  increases.  This  seems  to  support  a  hypothesis 
that  even  if  the  response  set  (therefore  the  database)  is  larger,  the  response-time  improvement  will  remain 
at  a  relatively  high  level  (between  40  an  50  percent). 
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Number 


Response-Time 

Improvement 

_ (A.B) _ 

*36.07 

45.14 

46.53 

48.94 

47.27 

48.81 

45.93 

47.79 

47.21 


Table  5.  The  Response-Time  Improvement  Between 
Configurations  A  and  B. 

Next,  we  calculate  the  response-time  reduction  for  each  of  the  nine  requests.  In  Table  6  we  present 
the  response-time  reductions  for  the  data  given  in  Table  4.  Notice  that  the  response-time  reduction  is 
worst  for  request  (I),  which  represents  a  retrieval  of  two  records  of  the  database.  On  the  other  hand,  the 
response-time  reductions  for  the  requests  which  access  larger  portions  of  the  database,  requests  (4)  and 
(6),  have  only  a  small  response-time  reduction.  In  general,  we  found  that,  the  response-time  reduction 
decreases  as  the  number  of  records  retrieved  increases,  i.e.,  the  response  time  remains  virtually  constant. 
Again  we  seem  to  have  evidence  to  support  the  hypothesis  that,  as  the  site  of  the  response  set  increases 
for  the  same  request,  the  response-time  reduction  will  decrease  to  a  relatively  low  level  (0.1%  or  less). 
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4.49 
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4.03 
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0.92 
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0.32 
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0.47 
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0.12 

7 

0.50 

8 

0.23 

9 

1.07 

Table  A.  The  Response-Time  Reduction  Between 
Configurations  A  and  C 

5.  CONCLUSIONS  AND  FUTURE  WORK 

We  have  shown  that  the  two  basic  performance  claims  of  the  multi-backend  databaae  ay  stem  are 
valid.  While  these  results  are  preliminary,  they  are  encouraging.  Overall,  the  response-time  improvement 
ranged  from  36.07  percent  to  48.94  percent,  when  the  number  of  backends  and  their  disks  is  doubled  for 
the  same  database.  The  low  end  of  the  scale  represented  a  request  which  involved  the  actual  retrieval  of 
only  two  records.  The  high  end  represents  a  request  which  has  to  access  all  of  the  database  information. 
The  response-time  reductions  were  also  impressive,  ranging  from  a  4.49  percent  change  to  a  0.12  change. 
In  other  words,  when  we  double  the  number  of  backends  and  their  disks,  the  response  time  of  a  request  is 
nearly  invariant  despite  the  fact  that  the  response  set  for  the  request  is  doubled.  Another  crucial 
discovery  that  we  made  was  that  the  results  were  consistent  and  reproducible.  The  tests  were  conducted 
at  least  twice  for  most  of  the  request  set,  with  the  testing  done  on  different  days  by  different  people.  The 
resulting  data  was  consistent  and  reproducible.  The  data  presented  in  this  paper  represents  the  last  set  of 
tests  for  the  request  set. 

The  next  logical  step  in  the  performance  evaluation  of  the  multi-backend  database  system  is  to 
extend  the  testing  to  include  the  other  request  types,  update,  insert  and  delete.  Additionally,  there  are 
still  some  more  tests  to  run  on  the  retrieval  request.  We  also  seek  to  provide  some  insight  into  the  inter¬ 
nal  performance  of  MBDS.  Internal  performance  measurement  provides  a  microscopic  view  of  the  system, 
by  collecting  the  times  of  the  work  distributed  and  performed  by  the  system  components,  i.e.,  in  our  case, 
individual  processes. 

Because  MBDS  is  intended  for  microprocessor- based  backends,  winchester-type  disks  and  an 
Ethernet-like  broadcast  bus,  we  will  not  continue  our  benchmark  work  on  the  present  VAX-PDPs  confi¬ 
guration.  Instead,  we  plan  to  download  MBDS  to  either  MicroVaxs  or  Sun  Workstations.  With  either 
choice,  we  can  utilise  a  broadcast  bus,  which  was  not  available  when  the  work  began  in  1981.  We  may 
also  eliminate  all  the  physical  and  virtual  memory  problems.  In  the  new  environment  we  can  perhaps 
obtain  a  more  thorough  benchmarking  of  MRDS,  and  study  various  benchmarking  strategies. 
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